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ABSTRACT 

^he  use  of  flight  simulators,  computer-based  instruction 
and  maintenance  training  simulators  for  training  is  evaluated 
on  the  basis  of  their  effectiveness  and  cost.  Flight  simula- 
tors are  cost-effective,  compared  to  the  use  of  aircraft,  for 
training;  so  are  maintenance  training  simulators  compared  to 
actual  equipment  trainers.  Computer-based  instruction  is  as 
effective  as  conventional  instruction;  comparable  cost  data 
are  not  yet  available,  so  one  cannot  say  whether  it  is  also 
cost-effective.  These  three  methods  of  training  are  not  more 
effective  than  the  methods  to  which  they  were  compared,  except 
for  small  improvements  in  a few  cases.  It  is  possible  they 
could  be  made  more  effective  if  cosi*,'s3v-ings  were  not  a major 
goal,  but  this  remains  to  be  determined.  The  goal  of  analyses 
of  training  should  be  an  ability  to  perform  trade-offs  of  the 
effectiveness  and  costs  of  new  methods! of  training,  but  no  such 
trade-offs  have  yet  been  made.  I 


PURPOSE 

The  purpose  of  this  paper  is  to  eval- 
uate both  the  cost  and  effectiveness  of 
military  training  in  three  areas  where 
relevant  data  are  available:  flight  simu- 
lators, computer-based  instruction,  and 
maintenance  training  simulators. 

THE  MAGNITUDE  OF  MILITARY  TRAINING 

Military  training  makes  a large  and 
continuing  demand  on  resources  allocated 
to  the  military  services.  For  example, 
the  time  spent  by  students  in  individual 
training  at  designated  schools  plus  that 
of  the  instructors  needed  at  those 
schools  accounts  for  about  one-fourth  of 
all  the  man-years  (both  military  and  ci- 
vilian) available  to  the  Department  of 
Defense  (Figure  1).  About  20  percent  of 
all  military  personnel  are  in  schools  at 
all  times,  either  as  students  or  instruc- 
tors. Most  (about  76  percent)  of  this 
effort  simply  provides  initial  training 
to  new  personnel  entering  military 
service  for  the  first  time. 

Individual  training  at  military 
schools  will  cost  S12.8  billion  in  Fiscal 
Year  1983.  The  types  of  training  vary 
widely  by  number  of  students  and  cost  per 
student,  as  shown  in  Figure  2.  This  in- 
formation can  help  tell  us  where  improve- 
ment in  training  could  have  large  impact. 
For  example,  specialized  skill  training 
involves  the  largest  number  of  students 
and  costs  more  than  any  other  type  of 
individual  training.  Undergraduate  flight 
training,  for  pilots  and  other  aircrew, 
has  the  smallest  number  of  students,  but 
it  costs  much  more  per  student  than  any 
other  type  of  individual  training. 

It  may  be  noted,  in  passing,  that 
student  needs  additional  training  after 


he  leaves  school.  This  is  known,  vari- 
ously, as  on-the-job  training,  advanced 
flight  training,  crew  training,  and  field 
exercises.  None  of  the  costs  of  addi- 
tional training  needed  after  leaving  a 
school  is  included  in  the  data  shown  in 
Figure  2. 

Finally,  it  may  help  to  know  that  the 
cost  of  individual  training  at  school  is 
nontrivial  when  compared  to  other,  readily 
available,  costs  in  the  annual  military 
budget.  As  shown  shown  in  Figure  3,  indi- 
vidual training  at  school  costs  about 
half  (or  more)  of  what  is  spent  for  intel- 
ligence and  communications,  the  strategic 
forces,  or  for  all  research,  development, 
test,  and  evaluation;  and  roughly  15 
percent  of  how  much  is  spent  for  procure- 
ment or  operation  and  maintenance. 

Military  training  is  a major  component  of 
the  military  budget.  Operational  readiness 
depends  on  effective  training.  It  is 
possible  to  realize  the  high  levels  of 
performance  built  into  our  advanced  weapon 
and  support  systems  only  if  these  systems 
are  maintained  and  operated  according  to 
their  design  specifications,  and  this 
also  depends  on  effective  training. 

Thus,  it  is  almost  mandatory  to  have 
effective  training;  improvements  in  our 
current  m°*\V'ds  of  training  are  likely  to 
have  usefu1  and  significant  payoffs. 

There  ic,  generally,  more  than  one 
way  of  training  most  skills,  e.g.,  group 
lectures,  individual  coaching,  use  of 
simulators,  and  so  on.  The  first  question 
which  must  be  answered — whether  it  con- 
cerns new  or  current  technology — is,  "Is 
it  any  good?  Does  it  train  as  well  as,  or 
perhaps,  better  than  some  other  way  we 
could  use?"  This  is  the  issue  of  effec- 
tiveness of  training.  But  the  effective- 
ness of  training  cannot  be  addressed  mean- 
ingfully without  also  considering  its 
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FIGURE  1.  The  Amount  of  Time  Spent  by  Students  and  instructors  in  Individual  Training 
at  Military  Schools,  FY  1983  (Source:  Military  Manpower  Training  Report  for 
FY  1983;  Weinberger  1982). 
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FIGURE  2.  Number  of  Students  and  Costs  of  Various  Types  of  individual  Training  at  Military 
Schools,  FY  1983  (Source:  Military  Manpower  Training  Report  for  FY  1983). 
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cosc.  Almost  any  method  of  training  could 
be  maae  defective  if  enough  money  for 
equipment  and  instructors  and  long  train- 
ing time  for  students  students  were  made 
available.  Therefore,  if  two  methods  of 
training  are  equally  effective,  we  should 
select  the  one  that  costs  less.  If  two 
methods  of  training  cost  about  the  same, 
we  should  select  the  one  that  is  more 
effective.  If  a new  method  of  training 
is  more  effective  and  costs  less,  the 
choice  is  easy  and  obvious,  but  such 
opportunities  are  rare. 

We  will  consider  next  comparisons  of 
the  effectiveness  and  cost  o-  three  types 
of  trailing:  flight  simulators  and  air- 
craft, computer-based  instruction  and 
conventional  instruction,  and  maintenance 
training  simulators  and  actual  equipment 
ttaint.-s.  Tnis  will  be  followed  by  a dis- 
cussioi  of  wi.at  the  results  mean  for  future 
evaluations  of  training. 

FLIGHT  SIMULATORS* 11 

The  extent  to  which  flight  simulators 
should  be  used  in  flight  training  is  a 
major  concern  to  all  military  services. 

Thu-  key  questions  are  obvious: 

o do  flight  simulators  really  train 
pilots 

o do  the  skills  learned  in  flight 
simulators  transfer  readily  to 
aircraf  t 

o are  flight  simulators  worth  what 
they  cost 

The  last  question  cannot  be  ignored  since 


*^This  information  is  based  on  Orlansky 
and  String,  1977. 
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the  cost  of  some  modern  flight  simulators, 
when  equipped  with  advanced  visual  systems 
and  motion  bases,  approaches  the  cost  of 
aircraft. 

The  procurement  of  new  simulators, 
including  major  improvements  to  those 
acquired  previously,  has  averaged  about 
$275  million  per  year  for  the  last  eight 
years  (Figure  4).  It  costs  about  $3.6 
billion  per  year  for  fuel  and  supplies 
needed  to  operate  military  aircraft. 

About  7,600  new  pilots  will  be  qualified 
in  fiscal  year  1983  at  a cost  of  about 
$1.8  billion. 

How  much  does  it  cost  per  hour  to 
operate  simulators  and  aircraft?  Figure  5 
shows  comparable  operating  cost  data  for 
42  pairs  of  flight  simulators  and  aircraft 
for  fiscal  years  1980  and  1981.  Variable 
Operating  Costs,  as  shown  here,  are  for 
fuel,  oil,  and  spare  parts  consumed  as  a 
function  of  use;  the  costs  of  pay  and 
amortisation  are  not  included  in  these 
amounts.  The  median  ratio  of  simulator/ 
aircraft  operating  costs  is  0.08;  it  was 
0.12  in  fiscal  years  1975  and  1976. 

The  fact  that  flight  simulators  cost 
less  to  operate  than  aircraft  is  a useful 
but  not  conclusive  finding  unless  we 
also  know  how  well  they  may  be  used  to 
train  pilots.  The  effectiveness  of  a 
flight  simulator  may  be  evaluated  by 
de  ermining  how  much  time,  if  any,  it 
saves  in  training  pilots  to  perform  speci- 
fic tasks  in  an  airplane,  compared  to 
the  amount  of  time  required  for  such  train- 
ing oply  in  the  airplane.  The  same 
level  of  performance  in  aircraft  is  re- 
quired in  both  cases.  It  has  long  been 
known  that  pilots  can  perform  in  the  air, 
more  or  less  well,  tasks  learned  in  flight 
simulators.  The  real  question  is  how  good 
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is  this  training.  An  index,  called  the 
Transfer  Effectiveness  Ratio  (TER),  shows 
the  amount  of  flight  time  saved  as  a func- 
tion of  the  amount  of  time  spent  in  the 
simulator  (Figure  6). 

Transfer  effectiveness  ratios  from  34 
different  studies  of  flight  training  (e.g., 
basic  contact  flight,  instrument  flight, 

ASW  maneuvers,  multi-engine  transition)  are 
shown  in  Figure  7.  The  TERs  range  in  value 
from  -0.4  to  1.9,  with  a median  value  of 
0.48.  Using  the  median  value,  the  data 
show  that  pilots  trained  in  simulators 
needed  less  time  in  aircraft  to  perform 
acceptably  than  pilots  trained  only  in 
aircraft;  the  amount  of  flight  time  saved 
was  about  one-half  of  the  time  spent  in 
the  simulator.  There  is  a wide  range  in 
these  TERs;  clearly,  these  flight  simu- 
lators were  not  equally  effective  in 
saving  flight  time  in  all  cases,  and  it 
seems  important  to  identify  the  factors 
that  make  for  efficient  and  inefficient 
uses  of  simulators. 

Since  flight  simulators  save  about  50 
percent  of  the  time  needed  by  pilots  to 
train  in  aircraft  and  cost  only  8 percent 
as  much  to  use  (median  values  in  both 
cases),  it  is  clear  that  flight  simulators 
are  cost-effective,  compared  to  the  use 
of  aircraft  alone  for  training.  The  amount 
of  such  savings,  after  providing  for  the 
cost  of  these  using  simulators,  is  shown 
for  three  cases  in  Figure  8;  the  savings 
were  large  enough  to  amortize  the  procure- 
ment cost  of  flight  simulators  within  two 
years  or  less. 

COMPUTER-BASED  INSTRUCTION ( 2 ) 

The  question  addressed  here  is  whether 
instruction  that  is  supported  by  the  use  of 
computers  is  cost-effective  compared  to 
conventional,  classroom  instruction.  Con- 
ventional instruction  is  based  largely  on 
lecture,  discussion,  and  tome  amount  of 
individual  coaching.  Its  calient  features 
are  group-pacing,  and  some  amount  of  indi- 
vidual attention.  Conventional  instruc- 
tion was  compared  to  two  new  methods  of 
instruction,  both  using  computers,  that 
have  been  developed  since  about  1960.  The 
first  method,  Computer-Assisted  Instruc- 
tion ( CAI ) , involves  placing  all  the  in- 
structional material  in  the  computer.  It 
provides  lessons  to  the  student  by  means 
of  a cathode  ray  tube;  the  student  may 
respond  by  such  means  as  a keyboard  or  by 
touching  the  screen.  The  computer  guides 
the  student,  corrects  him  as  needed,  and 
provides  new  or  explanatory  material  to 
suit  his  method  of  learning;  the  computer 
maintains  student  progress  records  and  can 
provide  a variety  of  administrative  infor- 
mation needed  to  operate  a course  and/or 
school.  In  the  second  method,  Computer- 
Managed  Instruction  (CMI),  the  student 
receives  his  instruction  away  from  the 
computer,  i.e.,  in  a learning  carrel  or  at 
a laboratory  bench.  However,  he  takes  a 
test  on  some  amount  of  course  material. 


'Based  on  Orlansky  and  String,  1979. 


typically  after  about  one  hour's  worth  of 
instruction.  The  computer  scores  the 
test,  interprets  the  results  for  the  stu- 
dent, and  directs  him  to  his  next  lesson 
or  to  repeat  the  last  one.  The  computer 
performs  a similar  evaluation  and  guidance 
function  m both  cases,  except  that  it  is 
based  on  individual  responses  in  CAI  and 
comes  only  after  tests  in  CMI.  The 
salient  features  of  CAI  and  CMI  are  that 
each  student  proceeds  at  his  own  rate  of 
learning  ("self-paced"  instruction)  and 
receives  frequent  feedback  on  his 
progress.  This  may  be  contrasted  with 
group-paced  instruction  where  tests  are 
spaced  further  in  time  (daily,  weekly  or 
monthly)  and  where  all  students  must  pro- 
ceed at  the  same  pace  despite  known  dif- 
ferences in  rate  and/cr  style  of  learning. 

Computer-assisted  and  computer- 
managed  instruction  are  significant 
alternatives  to  conventional  instruction 
primarily  in  specialized  skill  training. 
About  1.4  million  students  are  trained  in 
various  special  skills  at  a cost  of  about 
$2.9  billion  each  year  (Figure  9).  Most 
(76  percent)  of  these  students  are  "new 
accessions",  i.e.,  personnel  recently 
admitted  to  the  military  services  who  are 
being  trained  for  their  first  military 
assignment.  There  is  a continuous  need 
to  train  replacements  for  those  who  now 
leave  the  military  services  after  relativ- 
ely short  careers. 

CAI  and  CMI  may  have  many  advantages, 
but  the  fundamental  question  must  be  how 
effective  they  are,  compared  to  conven- 
tional instruction,  in  instructing  stu- 
dents. This  issue  has  been  addressed  in 
48  stud) es , conducted  from  1968  to  1979, 
that  are  summarized  in  Figure  10.  Using 
grades  in  end-of-course  tests  as  a measure, 
these  studies  compared  achievement  of 
students  in  the  same  courses  when  taught 
by  conventional  instruction  or  by  CAI  or 
CMI.  Six  different  CAI  systems  and  two 
different  CMI  systems  were  compared  in  a 
wide  variety  of  technical  training 
courses.  Student  achievement  with  CAI  or 
CMI  is  shown  in  the  figure  either  as 
"inferior",  "same",  or  "superior"  to 
achievement  with  conventional  instruction 
for  the  same  course;  the  result  of  each 
study  is  represented  by  a dot  placed  in 
the  appropriate  column.  The  figure  shows 
clearly  that  student  achievement  at  school 
with  CAI  or  CMI  is  about  the  same  as,  or 
in  some  cases  superior  to,  that  with  con- 
ventional instruction.  The  amount  of  supe- 
rior achievement  (i.e.,  higher  scores) 
is  not  large. 

In  conventional  instruction,  the 
rate  at  which  new  information  is  pre- 
sented is  based  on  about  how  much  the 
average  student  can  absorb.  The  entire 
class  proceeds  at  the  same  rate  (called 
"group-pacing"  or  "lock-step"  instruc- 
tion); this  pace  necessarily  holds  back 
students  who  can  learn  at  a faster  rate. 
With  CAI  and  CMI,  each  student  proceeds 
at  his  own  rate.  Thus,  we  should  expect 
that  students  instructed  by  CAI  or  CMI 
should  complete  their  courses  in  less 
time,  on  the  average,  than  those  taught  by 


FIGURE  6.  Transfer  Effectiveness  Ratio  (TER)  (Source:  Roscoe  1971,  1972). 
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FIGURE  7.  Transfer  Effectiveness  Ratios  for  Various  Types  of  Flight  Training,  Based  on  22  Studies  Performed 
1967-1977. 
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FIGURE  10.  Student  Achievement  at  School:  CAI  and  CMI  Compared  to  Conventional  Instruction. 
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conventional  instruction.  Data  on  student 
time  savings,  drawn  from  the  studies  on 
student  achievement  reported  above,  are 
shown  in  Figure  11.  The  average  amount  of 
student  time  saved  by  CAI  or  CMI,  compared 
to  conventional  instruction,  shows  a very 
wide  range — from  -30  percent  to  as  much  as 
85  percent  (i.e.,  in  a few  cases,  CAI  or 
CM  took  more  time).  The  median  value  is  a 
time  saving  of  about  30  percent. 

There  are  few  data  that  compare  the 
costs  of  CAI  or  CMI  and  conventional  in- 
struction for  the  same  course,  and  these 
comparisons  tend  to  be  incomplete.  Data 
on  student  time  savings  appear  to  be  reli- 
able; however,  many  other  costs  must  also 
be  considered,  e.g.,  for  program  develop- 
ment (hardware,  course  materials,  program- 
ming), the  delivery  of  instruction  (in- 
structors, support,  supplies,  repairs), 
and  student  pay  and  allowances.  No  such 
comparison  was  made  in  any  of  the  studies 
reported  above  that  contained  data  on  stu- 
dent achievement  and  time  saved. 

It  is  often  suggested  that  computer- 
based  instruction  is  cost-effective  be- 
cause the  cost  of  computers  has  declined 
dramatically  in  the  last  few  years.  This 
true  statement  can  be  misleading  because 
it  overlooks  the  fact  that  the  costs  of 
developing  software  and  course  materials 
(both  labor-intensive)  have  been  increas- 
ing over  the  same  time  period.  Some  stud- 
ies on  the  cost-effectiveness  of  the  PLATO 
IV  system  (at  Aberdeen  Proving  Ground, 

North  Island,  and  Chanute  Air  Force  Base) 
and  the  Air  Force  Advanced  Instructional 


System  (at  Lowry  Air  Force  Base;  were 
conducted  during  the  period  of  1975  to 
1978.  The  general  findings  were  that, 
compared  to  conventional  instruction, 
these  systems  were,  at  that  time,  either 
marginally  cost-effective  or  not  cost- 
effective.  None  of  these  studies  can  be 
represented  as  conclusive  or  used  as  a 
basis  for  judgment  at  the  present  time 
because  of  the  large  improvements  in  com- 
puter technology  that  have  taken  place 
since  they  were  completed. 

Thus,  it  can  be  said  that  computer- 
based  instruction  is  as  effective  as  con- 
ventional instruction  on  the  basis  of 
student  achievement  at  school,  and  that 
it  saves  student  time  needed  to  complete 
various  courses.  However,  comparisons  of 
the  total  costs  of  these  methods  methods 
of  instruction,  using  current  data  and 
technology,  remain  to  be  accomplished. 

MAINTENANCE  TRAINING  SIMULATORS* 3 > 

Operational  equipment  is  used  often 
at  schools  to  train  technicians  how  to 
perform  routine  maintenance,  diagnose 
malfunctions,  and  replace  faulty  parts. 

It  seems  reasonable  to  do  this  because 
trainees  will  have  to  maintain  the  same 
equipment  later  in  the  field.  In  addition, 
actual  equipment  needed  for  training  can 
be  acquired  readily  by  buying  an  additional 
unit  off  the  production  line.  Unfortu- 
nately, the  use  of  actual  equipment  is  not 

'^Based  on  Orlansky  and  String,  1981. 


FIGURE  11.  Amount  of  Student  Time  Saved  by  CAI  and  CMI,  Compared  to  Conventional  Instruction. 
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necessarily  the  most  effective  way  to 
train  new  technicians.  The  instructor  must 
be  able  to  insert  "faulty"  parts  in  order 
to  demonstrate  fault  finding  procedures. 

Apart  from  the  fact  that  it  takes  some 
time  to  prepare  each  lesson  on  faults, 
there  will  be,  at  best,  a limited  reper- 
toire of  faults  that  can  be  illustrated. 

Wear  and  tear  on  "failed"  parts  makes  them 
easy  to  detect  with  little  regard  to  the 
particular  malfunction  that  is  being 
demonstrated.  Major  faults  or  casualties 
are  not  introduced  in  large  systems  where 
the  investment  cost  is  high  for  fear  of 
damaging  the  trainer.  For  the  same  reason, 
only  the  instructor  makes  calibrational  or 
adjustment  changes.  Cascading  casualties 
are  difficult  if  not  impossible  to  induce 
into  real  equipment.  Further,  it  is  dan- 
gerous to  use  some  operational  equipment 
in  a school  because  of  high  voltages  or 
pressures;  operational  equipment  may 
break  down  at  school  because  of  abuse  by 
students,  not  be  ready  when  needed  for 
training,  and  may  entail  high  maintenance 
costs. 

Most  of  these  limitations  may  be 
overcome  by  using  maintenance  training 
simulators,  rather  than  actual  equipment, 
to  train  technicians.  Such  simulators 
always  incorporate  some  type  of  computer- 
based  control.  Thus,  it  is  easy  to  proviue 
a large  number  of  malfunctions  (e.g. , com- 
binations of  instrument  readings  and  non- 
responsive  controls)  that  an  instructor 
can  easily  and  quickly  select.  Simulators 
can  be  made  both  safe  for  use  and  resistant 
to  abuse  by  students.  Above  all,  being 
computer-based,  such  simulators  can  record 
student  performance  and  provide  knowledge 
of  results  that  otherwise  would  require 
fulltime  observation  of  every  student  by  a 
qualified  instructor.  The  use  of  mainte- 
nance simulators  is  increasing  gradually, 
primarily  in  aircraft  applications  (Figure 
12). 

The  effectiveness  of  maintenance  sim- 
ulators and  of  actual  equipment  trainers 
has  been  been  compared  in  about  15  studies 
conducted  from  1967  to  1980  (Figure  13); 
five  different  types  of  simulators  were 
evaluated.  The  findings  are  as  follows: 

1.  Simulators  were  as  effective  as 
actual  equipment  trainers. 

2.  Simulators  saved  some  student  time. 

3.  Students  favored  simulators  more 
than  instructors  did. 

Relatively  complete  data  on  the  acquisi- 
tion costs  of  maintenance  simulators  and  of 
comparable  actual  equipment  trainers  are 
available  in  11  cases.  Since  these  mainte- 
nance simulators  are  prototype  equipments, 
they  include  costs  both  for  development  and 
for  fabricating  a single  unit.  Therefore, 
two  estimates  were  developed  for  the  acquisi- 
tion cost  of  a maintenance  simulator:  a high 
estimate  that  includes  the  cost  of  research 
and  development  plus  the  cost  of  fabricating 
one  production  unit;  a low  estimate  that 
includes  only  the  cost  of  fabricating  one 
production  unit.  The  cost  of  an  actual 
equipment  trainer  is  the  recurring  produc- 
tion cost  plus  the  cost  of  modifying  that 
unit  to  make  it  useful  at  school. 


The  acquisition  cost  of  maintenance 
simulators  is  between  0.20  to  0.60  as  much 
as  that  for  comparable  actual  equipment 
trainers,  dep<  >ding  on  the  method  used  to 
estimate  the  cost  of  the  simulator  (Figure 
14).  The  lower  ratio  is  probably  the  more 
meaningful  comparison  because  it  excludes 
the  costs  of  research  and  development  both 
for  simulators  and  actual  actual  equipment 
trainers.  However,  a comparison  limited 
only  to  acquisition  costs  overlooks  the 
long  term  costs  of  using  simulators  and 
actual  equipment  trainers. 

Only  one  complete  cost-effectiveness 
evaluation  of  a maintenance  simulator 
and  its  comparable  actual  equipment  trainer 
has  been  found.  This  was  a 15-year  life- 
cycle  cost  comparison  of  the  Air  Force 
6883  Avionics  Test  Bench  3-Dimensional 
Simulator  for  the  F-lll  aircraft  and  the 
actual  equipment  trainer  (Cicchinelli, 
Harmon,  Keller  and  Kottenstette , 1980). 

The  study  finds  that  both  were  equally 
effective,  as  measured  by  student  perform- 
ance at  school  and  by  supervisors'  ratings 
of  performance  of  course  graduates  later 
on  the  lot).  Savings  due  tc  use  of  the 
simulators  are  due  primarily  to  lower 
operating  costs  over  the  period  of  inter- 
est. With  these  savings,  the  acquisition 
cost  of  the  simulator  can  be  amortized 
within  four  years  (Figure  15). 


DISCUSSION 

The  findings  of  these  studies  on  the 
cost-effectiveness  of  selected  types  of 
military  training  are  summarized  in  Figure 
16;  the  values  are  rounded  off.  The  over- 
all interpretation  must  be  that  innova- 
tions to  training  in  these  areas  are  as 
effective  as  the  methods  and/or  equipments 
to  which  they  were  compared.  Where  cost 
data  are  available,  the  innovations  cost 
less  to  procure  and  use  than  the  systems 
they  can  replace.  Thus,  research  and  de- 
velopment on  training  has  led  to  innova- 
tions that  are  cost-effective.  This 
overall  conclusion  must  be  qualified 
because  some  of  the  data  are  incomplete, 
as  discussed  next.  Credible  data  that 
compare  the  complete  costs  of  computer- 
based  instruction  and  conventional  in- 
struction have  not  yet  been  developed. 

This  applies  also  to  the  operating  and 
life  cycle  costs  of  maintenance  simulators 
and  of  actual  equipment  trainers.  There 
is  a need  not  only  for  more  complete  cost 
data  but  for  data  that  can  be  used  to 
identify  the  high  cost  elements  (i.e., 
cost-drivers)  in  such  training  systems; 
such  data  are  needed  also  to  develop  cost- 
estimating relationships  for  estimating 
the  probable  costs  of  new  training  sys- 
tems. 

It  is  important  to  note  that  the 
effectiveness  of  computer-cased  instruc- 
tion and  of  maintenance  simulation  has 
been  demonstrated  only  on  the  basis  of 
student  achievement  at  school.  Such 
data  do  not  tell  us  how  well  these  stu- 
dents perform  on  the  job,  and  job  perform- 
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HIGH  ESTIMATE 
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0.20 

FIGURE  14.  High  and  Low  Estimates  for  the  Acquisition  Costs  of 
Maintenance  Sfmulators/Actuai  Equipment  Trainers. 


AMORTIZATION 

ACQUISITION  COST 

$595  K 

SAVINGS  PER  YEAR 

$160  K 

$595 

AMORTIZATION = 

= 3.7  YEARS 

$160 

FIGURE  15.  Amortization  of  the  6883  Test  Stand,  Three-Dimensional 

Maintenance  Simulator  (Source:  Cicchinelli,  Harmon,  Keller 
and  Kottenstette,  1980). 
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ance  is  the  more  relevant  measure  of  train- 
ing effectiveness.  This  observation 
applies  equally  to  conventional  instruc- 
tion. One  exception  to  these  comments  is 
that  supervisors'  ratings  showed  about 
the  same  job  performance  for  students 
trained  either  with  the  6883  simulator  or 
actual  equipment;  however,  supervisors' 
ratings  are  not  objective  performance 
measures.  On-the-job  performance  measures 
are  an  inherent  aspect  of  evaluating  flight 
simulator  because  all  students  who  were 
trained  in  simulators  were  observed  later 
in  aircraft  in  flight,  as  were  obviously 
tnose  trained  only  in  aircraft.  However, 
measures  of  student  performance  in  air- 
craft are  based  on  instructors'  ratings 
on  well-developed  rating  scales  rather 
than  on  objective  measures  recorded  by 
instruments.  It  may  well  be  that,  regard- 
less of  method  of  training,  performance 
at  school  does  not  predict  later  perform- 
ance on  the  job.  Whether  there  is  a high 
or  low  degree  of  relationship  between 
measures  taken  at  school  and  later  on  the 
job  is  an  important  question  that  still 
remains  to  be  answered  for  almost  all 
types  of  training. 

A basic  premise  of  innovations  «.o 
training  such  as  flight  simulators,  compu- 
ter-based instruction  and  maintenance 
simulators,  is  that  they  will  improve  the 
quality  of  instruction  and,  thus,  the 
effectiveness  of  training.  The  basic 
findings,  as  reported  above,  are  that 
these  innovations  to  training  are  about  as 
effective  as  the  methods  they  can  replace 
and  save  appreciable  amounts  of  student 
time.  There  have  been  only  a few  demon- 
strations of  slight  improvements,  none 
large  enough  to  have  any  practical  signif- 
icance. Overall,  this  means  that  these 
innovations  yield  improvements  in  effi- 
ciency, i.e. , cost,  but  not  in  effective- 
ness. 

Innovations  to  training — based  on  what 
we  know  about  these  three  examples — may 
have  a potential  for  improved  effective- 
ness, but  this  has  not  been  conclusively 
demonstrated.  Superior  student  achieve- 
ment, compared  to  that  with  conventional 
instruction,  was  found  in  about  one-third 
of  the  studies  of  computer-assisted  in- 
struction ( CAI ) ; the  amounts  of  superiority 
were  small.  Two  such  results  (out  of  13 
cases)  were  reported  for  the  use  of 
maintenance  simulators;  none  for  flight 
simulators.  It  is  often  said  that  mainte- 
nance simulators  can  provide  better  train- 
ing because  they  can  demonstrate  a larger 
number  of  malfunctions  than  is  possible 
with  act'  al  equipment  trainers.  This 
plausible  claim  has  not  been  verified  nor, 
apparently,  even  tested.  Course  materials 
and  instructional  procedures  incorporated 
in  innovations  to  training  have  essentially 
been  modeled  on  the  current  methods.  This 
imposes  an  unnecessary  limitation  on 
improvements  m effectiveness  that  may 
well  be  present.  The  goal  of  almost  all 
improvements  to  training  so  far  examined 
seems  to  be  equal  effectiveness  and  savings 
in  the  cost  of  training;  only  lip  service 


has  been  given  to  improved  effectiveness. 

It  appears  possible  to  produce  improve- 
ments in  effectiveness  if  costs  were  kept 
constant,  i.e.,  if  large  savings  in  cost 
were  not  also  required  at  the  same  time. 

Finally,  there  has  been  no  attempt  to 
look  for  trade-offs  between  cost  and 
effectiveness.  To  do  so  would  require 
different  types  of  comparisons,  different 
experimental  designs,  and  a greater  concern 
with  improving  training  than  is  currently 
evident.  It  would  require  control  (in 
experiments) , over  the  amount  of  time 
students  spend  with  simulators  or  CAI/CMI 
to  determine  the  shape  of  the  learning 
curve  and  thus  the  optimum  time  to  go  from 
simulator  to  the  aircraft  or  actual  equip- 
ment. 

Additional  practice  in  training  brings 
diminishing  improvements  in  performance. 

The  well-known  phenomenon  of  the  learning 
curve  and  its  impact  on  the  cost-effec- 
tiveness of  training  has  been  almost 
totally  disgregarded  except  in  one  excel- 
lent study.  Povenmire  and  Roscoe  (1973) 
showed  that  the  transfer  effectiveness 
ratio,  which  measures  the  amount  of  flight 
time  saved  due  to  use  of  a simulator,  con- 
tinued to  decrease  and  eventually  became 
flat  (after  about  7 hours  in  this  particu- 
lar study  where  the  task  was  for  the  pilot 
to  qualify  in  a check  flight).  When  the 
decreasing  TERs  were  compared  to  the  costs 
of  using  the  simulator  or  aircraft  in  this 
case,  it  became  clear  that  it  would  be 
more  cost-effective  to  use  the  airplane 
rather  than  the  simulator  for  additional 
training  after  about  4 hours  in  the  simu- 
lator (and  not  after  7 or  11  hours,  as  was 
done  for  some  of  the  test  groups  in  this 
study) . 

A concern  with  trade-offs  between 
cost  and  effectiveness  would  also  require 
us,  in  effect,  to  develop  simulators  that 
differ  in  complexity,  and  therefore  in 
cost,  to  see  the  extent  to  which  effective- 
ness may  change  as  a consequence  of  changes 
in  the  cost  of  equipment.  Up  to  the  pre- 
sent, all  comparisons  have  been  made  with 
a simulator  designed,  more  or  less  to  an 
arbitrary  level  of  complexity,  as  the 
result  both  of  the  Instructional  System 
Design  process  and  o Z negotiations  between 
operators,  trainers,  ar.d  developers  over 
requirements  and  costs.  This  process 
needs  data  on  the  critical  relationship 
between  cost  and  effectiveness  in  order 
to  make  optimum  trade-offs. 


CONCLUSIONS 


Flight  simulators,  computer-based  in- 
struction and  maintenance  training  simu- 
lators appear  to  be  as  effective  as  the 
methods  of  training  they  can  replace;  they 
also  can  reduce  the  costs  of  training. 
Thus,  they  appear  to  be  cost-effective 
compared  to  more  conventional  methods  of 
training.  These  new  methods  of  training 
have  not  been  demonstrated  to  be  more 
effective  than  the  older  ones,  although  it 
is  possible,  if  cost  savings  were  net 
required  or  were  kept  constant,  that  they 
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might  be.  Trade-offs  between  the  effective- 
ness and  cost  of  alternative  training 
systems  have  not  been  performed.  To 
do  so,  more  reliable  and  more  complete 
data  on  effectiveness  and  cost  would  be 
required  in  most  areas  of  training. 
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